SemMT: A Semantic-Based Testing Approach for Machine Translation Systems

Machine translation has wide applications in daily life. In mission-critical such as translating official documents, incorrect can have unpleasant or sometimes catastrophic consequences. This motivates recent research on testing methodologies for machine systems. Existing mostly rely metamorphic relations designed at the textual level (e.g., Levenshtein distance) syntactic distance between grammar structures) to determine correctness of results. However, these do not consider whether original and translated sentences same meaning (i.e., Semantic similarity). Therefore, this paper, we propose SemMT, an automatic approach systems based semantic similarity checking. SemMT applies round-trip measures sentences. Our insight is that semantics expressed by logic numeric constraint be captured using regular expressions (or deterministic finite automata) where efficient equivalence/similarity checking algorithms are available. Leveraging insight, three metrics implement them SemMT. The experiment result reveals achieve higher effectiveness compared with state-of-the-art works, achieving increase 21% 23% accuracy F-Score, respectively. We also explore potential improvements achieved when proper combinations adopted. Finally, discuss a solution locate suspicious trip translation, which may shed lights further exploration.